Stream control failover utilizing an attribute-dependent protection mechanism

ABSTRACT

A technique for managing the streaming of digital video content to multiple clients involves identifying an attribute of a content element that is streamed to a client and selecting a protection mechanism for the content element as a function of the attribute, wherein the protection mechanism enables streaming of the content element to the clients in the event of a resource failure. In an example, the identified attribute is an indication of the popularity of the content element (e.g., as measured by the number of active streams), such that the protection mechanism is selected as a function of the popularity of the content element. In an embodiment, protection mechanisms that offer a higher level of protection are selected for the more popular content elements and protection mechanisms that offer a lower level of protection are selected for the less popular content elements.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is entitled to the benefit of provisional U.S. PatentApplication Ser. No. 60/667,300, filed Apr. 1, 2005, the disclosure ofwhich is incorporated by reference herein in its entirety.

This application is related to the co-filed applications entitled“STREAM CONTROL FAILOVER UTILIZING THE SHARING OF STATE INFORMATIONWITHIN A LOGICAL GROUP OF STREAM SERVERS” having application Ser. No.11/394,823, filed Mar. 31, 2006.

FIELD OF THE INVENTION

The invention relates generally to digital video networks that streamdigital video content to multiple clients, and more particularly, totechniques for stream control failover in such digital video networks.

BACKGROUND OF THE INVENTION

Digital video content can now be streamed to multiple clients inreal-time over traditional cable television and telephone networks, bothof which are being leveraged by service providers to provide moreattractive and varied services to customers. In order for a streamingvideo service to be commercially successful, it must be able to recoverfrom resource failures that will inevitably occur. To recover fromresource failures, streaming networks are typically equipped with backupequipment. In the event of a resource failure, all of the affectedstreams of digital video content are switched over to the backupequipment in the same manner. Switching all of the affected streams ofdigital video content over to the backup equipment in the same mannerresults in the all the streamed digital video content being protected tothe same level. Although this approach works well, as the quantity andvariety of streamed digital video content grows, the resources needed toprovide the same level of protection for all the streamed digital videocontent can become costly and may limit the ability of the network toscale up to support more streams and/or a greater variety of digitalvideo content.

SUMMARY OF THE INVENTION

A technique for managing the streaming of digital video content tomultiple clients involves identifying an attribute of a content elementthat is streamed to a client and selecting a protection mechanism forthe content element as a function of the attribute, wherein theprotection mechanism enables streaming of the content element to theclients in the event of a resource failure. In an example, theidentified attribute is an indication of the popularity of the contentelement (e.g., as measured by the number of active streams), such thatthe protection mechanism is selected as a function of the popularity ofthe content element. Typically, the amount of resources needed toprovide a protection mechanism is relative to the level of protectionprovided such that a higher level of protection requires more resourcesthan a lower level of protection. In an embodiment, protectionmechanisms that offer a higher level of protection are selected for themore popular content elements and protection mechanisms that offer alower level of protection are selected for the less popular contentelements. By intelligently selecting protection mechanisms thatcorrespond to the popularity of content elements, protection resourcescan be more efficiently utilized, thereby improving the scalability ofthe network.

Other aspects and advantages of the present invention will becomeapparent from the following detailed description, taken in conjunctionwith the accompanying drawings, illustrating by way of example theprinciples of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a network for streaming digital video content to multipleclients.

FIG. 2 depicts an exemplary matrix that correlates the number of activestreams of a content element to a particular protection mechanism.

FIG. 3A depicts the mapping of popularity statistics to priority.

FIG. 3B depicts the mapping of priority to protection mechanisms.

FIG. 4 depicts an exemplary mapping of different types of contentelements to priorities.

FIG. 5 depicts a process flow diagram of a technique for managing thestreaming of digital video content in the case where the protectionmechanism is selected at session setup.

FIG. 6 depicts a process flow diagram of a technique for managing thestreaming of digital video content in the case where the protectionmechanism for a content element is continuously evaluated.

FIG. 7 depicts an expanded view of one of the stream servers from FIG.1.

FIGS. 8A and 8B illustrate a protection mechanism that involves playingout the same digital video content to random access memory on twodifferent stream servers and switching the active interface in the eventof a resource failure.

FIG. 9 illustrates the sharing of state information amongst streamservers in the logical group of stream servers.

FIG. 10 illustrates a resiliency group that is established among asubset of the stream servers that make up the logical group.

FIG. 11 is a process flow diagram of a method for managing the streamingof digital video content to multiple clients.

FIG. 12 depicts an embodiment of a streaming network architecture thatsupports stream control failover in accordance with an embodiment of theinvention.

Throughout the description, similar reference numbers may be used toidentify similar elements.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 depicts a network 100 for streaming digital video content tomultiple clients. The network includes vault servers 102, a logicalgroup 104 of stream servers 106, a distribution network 110, clients112, and a backoffice module 114. The vault servers ingest digital videocontent from external sources and in the case of live television providedigital video content to the stream servers in real-time. Alternatively,the vault servers may store the ingested digital video content forsubsequent delivery to the stream servers, for example, in the case ofan application such as movies on demand. Vault servers that support anycombination of real-time streaming and storage of digital video contentare possible.

The stream servers 106 ingest digital video content from the vaultservers 102 and control the streaming of digital video content to theclients 112. The stream servers are combined into a logical group ofstream servers that communicate with each other at the Network Layerusing, for example, the Internet Protocol (IP). In an embodiment, thelogical group is established by associating each stream server that isto be included in the logical group with the same group identifier. Thestream servers in the logical group then communicate with the otherstream servers in the logical group using, for example, a multicast orbroadcast IP messaging protocol. As long as the stream servers cancommunicate with each other using a communications protocol such as IP,there are few, if any, physical boundaries to the logical group.

The distribution network 110 supports the distribution (e.g.,multicasting and unicasting) of digital video content from the streamservers 106 downstream to the clients 112. The distribution network alsosupports upstream unicast messaging from the clients to the streamservers and the backoffice module 114. The distribution network mayutilize any network technology that supports the distribution of digitalvideo content to multiple clients. In a packet-based environment, thedistribution network may utilize, for example, routers, switches,DSLAMs, cable modem termination systems (CMTSs), passive optical network(PON) architectures, or any combination thereof. In a hybrid fibercoaxial (HFC) environment, the distribution network may utilize, forexample, a combination of routers, switches, and QAMs. The distributionnetwork may also utilize wireless distribution technologies.

The clients 112 are systems that receive the digital video content fromthe distribution network 110 and provide the digital video content tovideo display devices (e.g., televisions). The clients may be embodiedas hardware, firmware, software, or any combination thereof and aresometimes referred to as set-top boxes (STBs). Clients may also beembodied as personal computers, home entertainment equipment, orintegrated into display devices. Clients in general are well-known inthe field.

Referring again to FIG. 1, multiple channels of digital video contentcan be provided to the stream servers 106 from the vault servers 102. Inthis embodiment, the digital video content that is provided to thestream servers is real-time digital video content received at the vaultservers and passed to the stream servers with little or no delay. Forexample, the digital video content includes real-time content from thecontent producers and/or distributors. The digital video content mayinclude, for example, live content (e.g., live sporting events, news,etc.) or pre-recorded content that is released by the content producersand/or distributors. In other embodiments, the digital video content maybe stored locally by the stream servers or the vault servers. Whetherthe digital video content is real-time or stored locally, the digitalvideo content is streamed to the clients via the stream servers.

A content element as used herein is any uniquely identifiable digitalvideo content. As an example, a content element may include alone or incombination: a titled content element (e.g., a titled program); a groupof titled content elements; a content sub-element (e.g., a portion of atitled program), a channel with which digital video content isassociated; an advertisement; a group of advertisements; a group ofpictures; a group of frames (e.g., a group of MPEG frames); and asequence of frames (e.g., a sequence of MPEG frames).

An attribute of a content element as used herein is any identifiablecharacteristic that is associated with the content element. As anexample, attributes of a content element may include alone or incombination: the popularity of a content element (e.g., as measured bythe number of active streams, the number of stream setup requests, therate of receipt of stream setup requests, the number of plays of acontent element in a given time period, and/or the number of requestsfor a particular start point of digital video content); the type ofcontent element (e.g., program content, advertisement content,subscriber communications, emergency alerts); the importance of thecontent element (highly important, moderately important, leastimportant). Note that a content element may include other attributesthat are not described herein and the scope of the attributes is notintended to be limited by the above-identified examples.

In order to provide reliable streaming of digital video content to theclients, it is important to be able to recover from resource failures.Examples of resource failures that may occur in a network that streamsdigital video content to multiple clients include, but are not limitedto: vault server failures; stream server failures; network interfacefailures; network connection failures; component failures (e.g., diskfailure, memory failure).

In the streaming network described with reference to FIG. 1, there arevarious different protection mechanisms that can be put in place torespond to a resource failure that affects the streaming of digitalvideo content to a client or clients. These protection mechanismstypically involve switching a stream from a failed resource (e.g., afailed stream server) to a resource (e.g., a different stream server)that is able to support the stream. In an embodiment, the amount ofresources (e.g., CPU, RAM, disk input/output (I/O), network I/O) that ittakes to respond to a resource failure is proportional to the desiredspeed of recovery. For example, a recovery mechanism that achieves veryfast recovery with no disruption of service on the client side isusually more resource intensive than a protection mechanism thatachieves a slower recovery and exhibits some disruption of service onthe client side.

In accordance with an embodiment of the invention, a technique formanaging the streaming of digital video content to multiple clientsinvolves identifying an attribute of a content element that is streamedto a client and selecting a protection mechanism for the content elementas a function of the attribute, wherein the protection mechanism enablesstreaming of the content element to the client in the event of aresource failure. In an example, the identified attribute is anindication of the popularity of the content element (e.g., as measuredby the number of active streams), such that the protection mechanism isselected as a function of the popularity of the content element.Typically, the amount of resources needed to provide a protectionmechanism is relative to the level of protection provided such that ahigher level of protection requires more resources than a lower level ofprotection. In an embodiment, protection mechanisms that offer a highlevel of protection are selected for the more popular content elementsand protection mechanisms that offer a lower level of protection areselected for the less popular content elements. By intelligentlyselecting protection mechanisms that correspond to the popularity ofcontent elements, protection resources can be more efficiently utilized,thereby improving the scalability of the network.

As mentioned above, one attribute of a content element that can be usedto select a protection mechanism is the popularity of the contentelement. In an embodiment, the protection mechanism for a particularcontent element is selected as a function of the current number ofactive streams. For example, the number of active streams associatedwith a streamed program is monitored and the protection mechanism forthe program is selected as a function of the number of active streamsfor the program. Applying this approach to multiple programs, the mostpopular programs can be given higher levels of protection than obscureprograms that are much less popular.

FIG. 2 depicts an exemplary matrix 118 that correlates the number ofactive streams of a content element to a particular protectionmechanism. In the matrix, the number of active streams of the contentelement is broken up into four ranges; 0-9, 10-249, 250-499, and 500 ormore. Each range is associated with a particular protection mechanism ormechanisms. As depicted in FIG. 2, the range of 0-9 active streams isassociated with protection mechanisms C and D, the ranges of 10-249 and250-499 active streams are associated with protection mechanism B, andthe range of 500 or more active streams is associated with protectionmechanism A. In an embodiment, the protection mechanisms are associatedwith the ranges such that the most popular content elements (e.g., thosecontent elements with the most active streams) are associated with theprotection mechanisms that afford the highest level of protection.Typically, the level of protection is a function of the speed with whicha stream can be switched from one resource to another (e.g., from afailed stream server to a working stream server). For example, thehighest level of protection is provided by a protection mechanism thatcan switch a stream to a different stream server in the event of aresource failure without any perceptible delay in the played out digitalvideo content and without losing a single frame of the digital videocontent. Lower levels of protection may involve some amount ofperceptible delay and/or frame loss during failover.

The protection mechanisms that are associated with an attribute of acontent element can be any protection mechanisms that are availablewithin the network. Additionally, the same protection mechanism can beassociated with different attributes, more than one protection mechanismcan be associated with an attribute, and the protection mechanisms don'tnecessarily have to be in order of increasing or decreasing level ofprotection.

In the embodiment of FIG. 2, the attribute of a content element isdirectly mapped to a protection mechanism. For example, the range of10-249 active streams maps directly to protection mechanism B. Inanother embodiment, the attribute of a content element is indirectlymapped to a protection mechanism. For example, FIG. 3A depicts a matrix120 mapping popularity statistics to priority and FIG. 3B depicts amatrix 122 mapping of priority to protection mechanisms. The combinationof the mappings depicted in FIGS. 3A and 3B creates an indirect mappingbetween the attribute of popularity and various protection mechanisms.The indirect mapping between an attribute and a protection mechanismprovides a level of abstraction that enables the protection mechanismassociations to be modified without reprogramming eachattribute/protection mechanism relationship. For example, referring toFIG. 3B, a modification of the priority/protection mechanism associationfor priority II will in turn affect both the 10-249 and the 250-499active stream ranges depicted in FIG. 3A. Although in FIGS. 3A and 3B,the popularity is mapped to a priority, the priorities could be someother set of categories, which are then mapped to protection mechanisms.

In another embodiment, the attribute of the content elements used toselect the protection mechanism is the type of content element ratherthan the popularity of the content element. FIG. 4 depicts an exemplarymatrix 124 mapping different types of content elements to priorities. Asindicated in FIG. 4, the types of content elements include program-gold,program-silver, program-bronze, advertisement-gold,advertisement-silver, advertisement-bronze, and emergency alert, wherethe gold, silver, and bronze designations relate to service levels. Thecorresponding priorities of the different types of content elements canthen be associated with protection mechanisms as described withreference to FIG. 3B.

The selection of a protection mechanism for a content element can takeplace, for example, at session setup and/or on a continuous basis. FIG.5 depicts a process flow diagram of a technique for managing thestreaming of digital video content in the case where the protectionmechanism is selected at session setup. At block 130, a session setuprequest for a content element is received. At block 132, an attribute ofthe content element is identified and at block 134, a protectionmechanism for the content element is selected as a function of theattribute, wherein the protection mechanism enables streaming of thecontent element in the event of a resource failure.

FIG. 6 depicts a process flow diagram of a technique for managing thestreaming of digital video content in the case where the protectionmechanism for a content element is continuously evaluated. After aninitial start process 136, at decision point 138, it is determinedwhether or not the status of the content element should be evaluated. Inan embodiment, the status of a content element is evaluated on aperiodic basis although other triggers are possible. For example, thenumber of active streams associated with a program or channel can bedetermined at a regular time interval (e.g., once a minute). If thestatus of the content element is not to be evaluated, then the processreturns to decision point 138. If the decision is to evaluate the statusof the content element, then at block 140, an attribute of the contentelement is identified. For example, if the attribute of the contentelement is popularity, the current number of active streams associatedwith the content element is determined. Once the attribute isidentified, at decision point 142, it is determined if the protectionmechanism matches the identified attribute. For example, it isdetermined if the protection mechanism that is in place corresponds tothe current number of active streams for the particular content element.If the protection mechanism matches the identified attribute, then theprotection mechanism remains unchanged and the process returns todecision point 138. On the other hand, if the protection mechanism doesnot match the identified attribute, then at block 144 a protectionmechanism that matches the identified attribute is selected. Forexample, the protection mechanism can be adjusted to a higher or lowerlevel of protection depending on the current number of active streamsfor the content element. Once the new protection mechanism is selected,the process returns to decision point 138.

FIG. 7 depicts an expanded view of one of the stream servers 106 fromFIG. 1. The stream server depicted in FIG. 7 includes a session setupmodule 150, an attribute engine 152, a protection mechanism selector154, and a state management module 156. The session setup moduleprocesses session setup requests that are received from the backofficemodule 114, the clients 112, and/or other stream servers. The sessionsetup module communicates with the protection mechanism selector toidentify a protection mechanism for a stream session. In the embodimentof FIG. 7, the attribute engine manages the identification of anattribute that is associated with a content element. For example, if theattribute is the number of active streams, the attribute enginedetermines how many active streams are associated with the contentelement. If the attribute is the type of content element, the attributeengine includes an association between content elements and contentelement types. The protection mechanism selector utilizes informationfrom the attribute engine to select the appropriate protection mechanismfor a particular content element. For example, if the attribute is thenumber of active streams, the protection mechanism selector obtains fromthe attribute engine the number of active streams for a content elementand translates the number of active streams into a protection mechanismas described with reference to FIGS. 2-6.

The selection of the protection mechanism for a particular contentelement can be made, for example, on a stream-by-stream basis (e.g., foreach instance of the content element). For example, different protectionmechanisms can be selected for different streams of the same titledcontent element. This can be used, for example, to provide differentqualities of service for different subscription levels.

In an embodiment, the popularity of a content element is determined on alogical group basis. However, in other embodiments, the popularity of acontent element can be determined on a stream server-by-stream serverbasis or a more global basis than the logical group.

As described above, various different protection mechanisms can be usedto recover from a resource failure. In accordance with an embodiment ofthe invention, a protection mechanism, referred to herein as “hotstandby,” is described with reference to FIGS. 8A and 8B. Referring toFIG. 8A, two stream servers 106 each include an ingest module 160,random access memory (RAM) 162, and an interface (I/F) 164 that connectsthe stream servers to the distribution network. According to the hotstandby protection mechanism, the same digital video content 166 isprovided to both of the stream servers (i.e., to stream servers A andB). Both of the stream servers ingest the digital video content andstore the digital video content, at least temporarily, in theirrespective RAM. As is known in the field, digital video content istypically played out to RAM before it is transmitted onto a network linkthrough an interface. Typically, the RAM stores a small moving window ofthe digital video content, simultaneously receiving new frames ofdigital video content and discarding old frames of digital video contentsuch that the amount of digital video content stored in the RAMmaintains a steady state. As depicted in FIG. 8A, one of the streamservers (i.e., stream server A) transmits the digital video content thatis stored in RAM out its interface to the clients. The other streamserver (stream server B) continues to play the digital video content toits RAM but does not activate its interface unless or until a failure isdetected in stream server A. If a failure is detected in stream serverA, because stream server B has the digital video content played out toits RAM, the stream server can begin transmitting the digital videocontent by simply activating its interface. FIG. 8B depicts the digitalvideo content being streamed to the client via stream server B insteadof stream server A after a failure is detected in stream server A. In anembodiment, the interface of stream server B is activated fast enoughthat there is no perceptible delay of the displayed digital videocontent and no dropped frames. Although RAM is described herein as thetemporary memory, other types of high speed memory such as Flash memorymay be used to temporarily store the digital video content before it istransmitted out the network interface. Additionally, although not shownthe stream server typically includes hard disk memory that allows thestream server to locally store, or cache, ingested digital videocontent.

In an embodiment, protection mechanisms are broken down into threelevels. The highest level of protection is achieved by the hot standbytechnique described above, a middle level of protection is achieved byensuring that a content element is locally cached at a stream server sothat it can be quickly accessed (although not as quickly as hotstandby), and a lowest level of protection that does not specify eitherhot standby or local caching but allows the content element to be storedin any convenient manner.

As described above, one attribute of a content element can be theimportance of a content element relative to other content elements. Forexample, a content element that is associated with an emergency alertmay be deemed to be more important than a content element associatedwith an advertisement. In an embodiment, a protection mechanism thatprovides a higher level of protection is selected for a more importantcontent element and a protection mechanism that provides a lower levelof protection is selected for a less important content element.

In an embodiment, the resource requirements of the different availableprotection mechanisms are identified. Further, a fault recoveryrequirement is established for a content element. For example, the faultrecovery requirement indicates a maximum acceptable delay and/or frameloss for a content element. With this information available, protectionmechanisms are selected by finding the protection mechanism with thelowest resource requirement that still meets the fault recoveryrequirement for the particular content element. Using this technique,the level of protection provided by a protection mechanism is matched tothe particular fault recovery needs of the content element, therebypreventing resource intensive protection mechanisms from beingunnecessarily applied to content elements that can be satisfactorilyprotected by less resource intensive means.

Some streaming networks support trick modes (e.g., rewind, fast rewind,pause, fast forward). In an embodiment, the same protection mechanismsthat are applied to the original content elements are applied also tothe associated trick mode objects. However, an in alternativeembodiment, protection mechanisms are applied to trick mode objectsindependently of the original content elements. This allows differentlevels of protection to be applied to a content element and itsassociated trick mode objects.

In addition to selecting protection mechanisms, other operations can beperformed as a function of an attribute of a content element. Forexample, streams of the same content can be distributed amongst streamservers in a logical group as a function of popularity. In oneembodiment, streams of a more popular content element (e.g., a programor a channel) are distributed more widely amongst the stream servers. Arelatively wide distribution of streams amongst the stream servers canmake it easier and quicker to switch streams from one stream server toanother in the event of a resource failure.

In an embodiment, the attribute of a content element is identified inreal time and reflective of current network conditions. For example,popularity statistics are dynamically computed based on actual demandfor content elements within the logical group of stream servers.

In accordance with another embodiment of the invention, a technique forstreaming digital video content to multiple clients involvesestablishing a logical group of IP-connected stream servers, generatingstate information at multiple stream servers within the logical group,sharing the state information amongst stream servers in the logicalgroup, and using the shared state information to respond to resourcefailures within the logical group. By sharing state information withinthe logical group of stream servers, stream servers with the sharedstate information can quickly take over the active streams from a failedstream server within the logical group. The quick transition ofresponsibility from one stream server to another enables the streamingnetwork to provide continuous streaming of digital video content to theviewer without interruption. Further, the sharing of state informationbetween stream servers in the logical group allows the stream servers tosupport active streams while also serving as backup stream servers forthe logical group. This avoids the need to have dedicated backup streamservers that sit idle until a resource failure occurs.

FIG. 9 illustrates the sharing of state information 166 amongst streamservers 106 in the logical group 104 of stream servers. In theembodiment of FIG. 9, each of the stream servers supports active streamsand generates state information related to the active streams. That is,none of the stream servers is dedicated solely to a backup function.Various techniques can be used to share state information amongst thestream servers in the logical group. In one embodiment, stateinformation is copied from a state machine and distributed to the otherstream servers in the logical group. In another embodiment, streamservers maintain dedicated state machines for the other stream serversand the stream servers share information that is input to the statemachines (referred to herein as state machine inputs). The state machineinputs are then provided to the respective state machines that aremaintained for the different stream servers. In an embodiment, in orderto ensure quick and accurate switching of streams between stream serversin the logical group, state information is shared between the streamservers in real-time. State information can be shared amongst the streamservers in the stream group using, for example, a unicast floodingprotocol.

In an embodiment, a resiliency group is logically established as asubset of the logical group of stream servers and state information isshared within the resiliency group instead of within the entire logicalgroup. FIG. 10 illustrates a resiliency group 170 that is establishedamong a subset of the stream servers 106 that make up the logical group104. FIG. 10 also illustrates the sharing of state information 166between the stream servers in the resiliency group. The use of aresiliency group allows the sharing of state information to be morelimited. This allows certain stream servers in the logical group to beresponsible for the failover control and limits the volume of stateinformation messaging within the logical group.

A technique for managing the streaming of digital video content tomultiple clients is described with reference to FIG. 11. At block 180, alogical group of IP-connected stream servers is established. At block182, state information is generated at multiple stream servers in thelogical group. At block 184, state information is shared amongst streamservers in the logical group. At decision point 186, it is determinedwhether or not a resource failure has been detected. If no resourcefailure has been detected, the process returns to block 182, where stateinformation continues to be generated by the stream servers in thelogical group. On the other hand, if a resource failure is detected, atblock 188, the state information is used to switch an active stream to adifferent stream server in the logical group. After switching the activestream to a different stream server in the logical group, the processreturns to block 182, where state information continues to be generatedby the stream server in the logical group.

In addition to using the state information for failure protection, theshared state information can be used to manage resources within thelogical group 104, including, for example, assigning streams to streamservers 106 in the logical group, servicing stream requests, and loadbalancing.

Referring back to FIG. 7, the state management module 156 is responsiblefor managing the generation of state information, the sharing of stateinformation amongst stream servers 106 in the logical group 104, and theusing of the shared state information to respond to resource failureswithin the logical group.

As used herein, the terms “multicast” and “multicasting” refer to atechnique for providing the same digital video content to multipleclients in which the digital video content is delivered over commonlinks only once (e.g., the digital video content is copied when itreaches nodes with links to multiple destinations). As used herein,multicast and multicasting are synonymous with the terms broadcast andbroadcasting as related to, for example, hybrid fiber coaxial (HFC)cable networks. As used herein, the terms “unicast” and “unicasting”refer to a technique for providing digital video content to a singlespecified client.

In some applications, the distribution network includes a packet-basednetwork. In packet-based networks, multicasting may involve replicatingpackets at nodes that include multiple branches leading to differentclients. The replication of packets at branching nodes eliminates theneed to send multiple packets of the same content over the same link.Packet-based distribution networks may utilize, for example, IP,Ethernet, ATM, or a combination thereof to communicate digital videocontent. In packet-based networks, unicasting typically involvespoint-to-point messaging between nodes (e.g., servers and clients).Point-to-point messaging can be accomplished, for example, usingwell-known source/destination address based protocols (e.g., IP orEthernet).

In some applications, the distribution network includes an HFC networkthat utilizes radio frequency signals (RF) for local distribution ofdigital video content to the clients. In HFC networks, multicastingtypically involves distributing all channels to all clients. Each clientis able to receive any channel by tuning to the desired channel. In HFCnetworks, unicasting may involve distributing a channel, which isintended for only one client, to multiple clients and coordinating withthe intended client so that only the intended client is able to receiveto the desired channel. Even though the channel may be distributed tomultiple clients, only one client, the intended client, is able toaccess the channel and display the digital video content. For purposesof this description, a communications technique such as this, which canbe implemented in HFC networks, is considered unicasting.

As is known in the field, the streams servers depicted in FIGS. 1 and 7include a processor and memory. The processor may include amultifunction processor and/or an application specific processor that isoperationally connected to the memory. The processor performs functions,such as executing software code, which are well-known in the field. Thememory within the channel change server and the client may includecircuits for storing processor-executable instructions, for bufferingdigital video content, and for storing data structures. Although theprocessor and memory are depicted as separate functional units, in someinstances, the processor and memory are partially or fully integratedonto the same device. Processor-executable instructions generallyinclude programs, protocols, data structures, routines, objects, andinterfaces that can perform and/or enable certain useful functions. Inan embodiment, the attribute engine, the protection mechanism selector,and the state management module are embodied as processor-executableinstructions that are stored in memory that is resident on the streamservers.

The above-described techniques for provide stream control failover canbe applied to any type of network that is able to stream digital videocontent to multiple clients.

FIG. 12 depicts an embodiment of a network 200 that supports streamcontrol failover in manner that provides continuous media streaming to aclient with minimal or no delay and/or loss of frames. The networkincludes a content vault 202, a logical group 204 of stream pumps 206, abackoffice 214, and a client 212. Although only one client is depictedfor description purposes, the network includes multiple clients as isknown in the field of streaming networks. One of the stream pumpsincludes a stream service master (SSM) 290 and a stream control proxy292, and each of the stream pumps includes a stream service slave (SSS)294, and a stream control server (SCS) 296. The backoffice includes asession setup module 298 and the client includes session setup andsession control modules 300 and 302, respectively. Functions of theabove-identified elements are described below. The stream pumps mayalternatively be referred to as stream servers and the content vaultsmay alternatively be referred to as vault servers. Further, the elementsdepicted in FIG. 12 are related to the corresponding elements describedwith reference to FIG. 1.

A. Terminology

-   SSM Stream Service Master-   SCP Stream Control Proxy-   SCS Stream Control Server-   SSS Stream Service Slave-   SCC Stream Control Client (e.g. settop box, STB)-   Streaming Pump Streaming pump with cached real time stream (cached    in RAM and optional disk)-   Backoffice Client Control plane software for session setup and    teardown-   STB Client Software in device like STB for dynamic control of stream    session-   Failover-IP The IP address exported to external components such as    Backoffice Control and Settop Box. One Failover-IP exists per pump    group.-   Internal-IP One or more unique IP addresses of each physical    platform.-   Topology Steering Redirecting a session from one streaming pump to    another streaming pump based upon physical proximity of downstream    networking resources. In cable networks for example, QAM service    groups are assigned a routing affinity to streamers, and, in some    configurations are constrained in downstream physical connectivity    (i.e. unreachable by all but a subset of the streamers).-   Resource Allocation Function performed by the SSM which evaluates    information such as load balancing, actual cached contents per    stream pump, and downstream topology affinity to choose the current    best stream pump for handling this stream.-   ESH External stream handle (ESH) is assigned by the streaming system    and exposed to external signaling elements such as backoffice    session manager and client session controller.-   ISH Internal Stream handle (ISH) is assigned by the streaming system    but retained internally only. The ISH and ESH have a 1-to-1 mapping    for each and every stream.-   Shared State Shared state information propagates in near real time    between stream servers and vault servers or relevant subsets of the    nodes.-   Downstream Affinity Configured or discovered information as to the    downstream connectivity for each streaming server.-   Local Cache Each node in a logical stream server group has a local    cache of streams. The streams may be located in RAM or in disk or    fragments in both. All or a part of a stream may be in cache.-   Callback Identifier An internal interface that allows one process or    function to invoke another function, e.g. a callback function for    handling stream session control operations (e.g. termination of    stream).-   Queued Play A method of organizing the stream playout from    independent time segments of an existing stream.-   Provisioned Multiple A method of organizing the stream playout by    serially linking multiple independent streams together in time.-   Trickle Down Mode A method of taking a streamer out of service by    continuing to play active streams until they are terminated or    transferred to other streaming services, while not initiating any    new streams on the server.-   GOID Global Object Identifier. Uniquely identifies an object in the    cluster of servers. Multiple objects, each identified by their GOID,    are configured for each stream.

The stream pumps 206 are organized into a fault tolerant logical group204. Any method can be used by the group members to discover eachother—static configuration, dynamic detection through the shareddatabase.

B. Stream Pump Master

A heartbeat protocol is executed dynamically between the members of thestream pump logical group 204. One of the stream group members iselected as the master. Two functions are activated within the master.

1. Stream Service Master (SSM)

The SSM 290 is a process that is used to allocate all resourcesassociated with a stream session in order to deliver the streamsuccessfully to the client 212 (e.g. to a set top box). It uses loadbalancing information, cached contents per streamer, QAM steering, anddownstream affinity to select the best suitable streamer within alogical stream server group. It also keeps track of all allocated andavailable resources for each individual stream server 206 within alogical stream server group.

2. Stream Control Proxy (SCP)

The SCP 292 forwards real time stream control commands (e.g. play,pause, resume, stop, fast-forward, rewind, and jump etc) to theappropriate stream server 206 within the logical group 204 even afterthe stream is handed over to another stream server in the case, forexample, where the first stream server has its power completely turnedoff. The transition between stream servers is totally transparent,seamless, and glitchless to the stream control client (e.g. set top box)which is receiving and controlling its stream.

Each node in the stream group has two IP addresses. The failover IPaddress is an address that is assigned to at most one node within thegroup. The failover IP address is assigned to the stream server thatincludes the SSM/SCP functions. The failover IP address is visible toexternal clients, for example, both the session control client and thestream control client. The Internal IP address is the address of thenode itself. It is not exposed to external clients.

When the heartbeat function detects that the master stream server hasfailed, a new stream service master is elected. The failover IP addressis activated by the new master stream server. Active TCP connectionswill be broken and the clients will reestablish the TCP connection, butto the new master stream server. Sessions will continue uninterrupted.

After activating the failover IP address the master node listens forincoming TCP connection requests from session control and stream controlclients (and any other client programs that can communicate with thestreaming group). The master also starts both the SSM and SCP processes290 and 292. The SSM and SCP processes discover all information aboutactive streams, current load, etc from locally stored information andfrom the shared state information.

Stream Session Handles

In an embodiment, each active streaming session has two handles—theexternal stream handle (ESH) and the internal stream handle (ISH). Whena stream is first created the ESH and ISH are identical. If the streamremains on the same stream server the handles continue to be equal. Ifthe stream is moved to another stream server for any reason, the ESHstays the same for external clients, but the ISH is redefined toidentify the stream on the new stream server. The SCP is responsible formapping the ESH to the ISH for messages to and from external clients.

The streaming content-library and streaming system exposes the externalresource-handle to external backoffice signaling control functions.Internally the system provides a dynamic proxy mapping function in orderto deliver fault tolerance to external control functions. Three physicalinterfaces into the system—stream session setup (e.g. BMS), dynamicstream control (e.g. LSCP to STB), and content store control. In eachinterface, an external version of the resource handle (e.g.session_handle) is preserved, regardless of the location of the internalserver. A proxy interfaces with the external control function. The proxycreates a mapping between the external resource-handle and the actualinternal resource handle. The proxy and internal servers maintain a realtime mapping between the two handles using an internal shared databasedistribution. From the perspective of the external control function, asingle node address is presented (virtual IP address of the proxy) and asingle resource handle is provided (mapped by the proxy to an internalhandle).

C. Stream Service Slave Nodes

In an embodiment, all nodes in the logical group 204, both the masternode and the slave nodes, have two processes that are always active.Both processes create connections to the stream service master usingtheir internal IP address. They register themselves to the SSM 290 andSCP 292 when they bring up the connection. This also allows the SSM andSCP to initiate the connection using information gleaned from the sharedstate information 266 and other locally stored information.

1. Stream Service Slave (SSS)

The SSS handles allocation of resources for the stream as requested bythe SSM.

2. Stream Control Server (SCS)

The SCS handles all dynamic stream control commands that are forwardedto it by the SCP 292. As part of allocating the resources for a stream,the SSM 290 locates one of the SSS 294 based on the criteria which canserve that stream the best. The SSM then allocates the stream to thatstream server and sends the stream initiation request to the SSS withinthat stream server. The SSS shares the stream handles, and memory tocommunicate with caching systems. Every time the SSS comes up, itsynchronizes with its local caching system and becomes aware of alllocal streams contained within the local caching system.

The SCS 296 receives stream control commands from the SCP 292. The SCPopens a new connection to the SCS for each client connection it has. TheSCP checks the encoded stream handle and finds out where the controlcommand has to go. Once the SCS gets the commands from the proxy, itvalidates the stream handle and passes it to the local caching systemwith a callback (or equivalent internal linkage) for it to take theaction for the stream associated with the handle.

The SCS 296 keeps track of the states of each stream and each call backidentifier it has passed to the local caching system. The local cachingsystem calls the SCS back after the stream is over, erred, switched tothe next queued play command, switched to the next content in case ofprovision multiple, and initiation of playing of live streams.

The SCS sends back a proper response to the SCP after processing thecontrol commands it gets from the SCP. After receiving the response, theSCP then forwards the response to the correct external client.

There are some asynchronous responses from the local caching system(e.g. the stream is over) which are also sent to the client. Theseasynchronous responses are also received by the SCS, sent to the SCP,and then forwarded to the correct external client.

D. Stream Handover Method

A stream can be handed over from one stream server to another streamserver based on any of several criteria: load balancing, server crash,or power failure.

In a stream server that has the handed over stream, the local cachingsystem passes all information about the stream to the SCS through thecallback function. The original stream handle, the stream handleassociated with stream initially called as external stream handle, andthe new internal stream handle which is being used by the new streamserver is communicated to the SCS through the callback mechanism. TheSCS then delivers this information to the SSS, SSM, and SCP toseamlessly handle the all inbound actions for that stream.

E. Additional Features

In addition to the basic feature of fault tolerance there are additionalenhanced features that strengthen the resiliency and scalability of themechanism.

1. Trickle Down Out of Service Mode

A load balancing protocol for allocating new streams on a specificstream server in a logical group can be configured to help the SSM totrickle down any stream server in the logical group. The trickle downprocess enables the SSM not to allocate any new streams to the streamserver which has been trickled down, but the old streams still keepgoing. In this way, the stream servers could have zero down time forsystem upgrade.

2. Topology Steering

There are situations in which the actual downstream topology isdifficult if not impossible to compute until later in the stream sessionsetup phase. There are also fault tolerance situations in which adownstream network resource is taken out of service or fails. In eitherof these cases, a stream may need to be moved to a different streamserver. This feature is called Topology Steering (or QAM steering forCable). The failover mechanisms described herein are used to assure aseamless transfer of the stream from one stream server to another streamserver with better topological affinity.

In an embodiment, topology steering is invoked based upon load. If astream server is congested, a second tier stream server, one in the samelogical group or one further upstream in a regional node, is assigned toplay out the stream.

3. Real-Time Communication of Vault Properties

An important attribute that is consulted as part of the algorithm todecide whether or not to retain and reuse cached content is the set ofproperties for the content stream. This attribute boosts the performanceof allocating streams on each stream server. The ingestion function ofthe content vault dynamically tracks these properties and dynamicallycommunicates changes in properties to all stream servers using theshared state information. Attributes of an individual vault contentstream may include, for example, content-added, content-deleted, andcontent-changed.

4. Popularity Attribute Computation

The servers dynamically compute popularity of content based upon actualdemand. The popularity rating may be computed locally or may becommunicated to other servers and computed globally. In an embodiment,the servers use the popularity information to prioritize requests todelete content in order to free up cache space.

The popularity of content can influence the choice of resiliencyalgorithm ranging from hot standby to cold standby to ensure thequickest and most seamless failover mechanism for the most popularcontent elements.

5. Hot Standby Streaming

For a certain set of streams that have been declared high priority andpopular, a mechanism is defined which allows the streams to betransferred to another stream server without losing a frame. The streamis cached and played out into RAM on a mirroring stream server. But themirroring server does not transmit the stream over its networkinterfaces. When the failover is detected the stream is played out fromRAM, and the client sees no degradation at all.

6. Load Distribution Resiliency

A stream consists of multiple objects, each identified by its globalobject identifier (GOID). A stream (can also be called a “title” or“content”) consists of the normal play object, its trick mode objects,and an igate object that contains meta-data information (like where theI-frames are) for the content. In addition there can be a delta filethat tracks changes that have been made to the normal play object sothat it is possible to recreate the original content for ftp out (wherethe system acts as a content server for other VOD servers).

Two variables measuring “business” or load are defined. One is theamount of a GOID stored on the local disk and the other is a busy levelfor the local CPU resources. These “busy” (CPU load and bandwidth)measures can to be made relative either by including max capability inthe cost or exporting relative cost in these dimensions so decisions canbe made across systems of varying capability.

Two parameters are defined to enhance resiliency for the loaddistribution function. The first is a basic threshold to decide whethera stream has enough usage to worry about resiliency. For maximumresiliency this threshold is set to zero. For less resilientconfigurations it can be set higher. A typical setting would be so onlystreams which had been used over 10's to 100's of times would bedistributed for resiliency. The other parameter is a limit on thedynamic range of load on a given GOID across stream group members. Thisis important to increase the probability of being able to failover intothe RAM cache. A default value is such that the ratio of minimum load tomaximum load between stream server in a logical group is the size of thelogical group. That is if the group size is four then the ratio ofminimum load to maximum load on a particular GOID is not allowed toexceed four. This combined with the locality measurements forces load ona particular GOID to be distributed about a logical group in “clumps”.The limit on dynamic range stream group member would already have loadon all popular titles so failover would be less difficult. This allowsfor both reasonable cache efficiency and quick recovery from failure.

One factor affecting the programmed delay introduced into stream playouthas to do with latency needs of the client. If resumption of the streamwill be delayed due the need for local or remote disk access in theclient, an appropriate delay will be introduced, and the stream will bepadded with fake frames by the stream server. Once the stream is resumedat the client a second delay parameter is the delay at which a splice isintroduced rather than a simple resumption of the stream.

7. Optimizing Speed, Distribution, and Load Balancing

The three key variables governing stream recovery algorithm are streamrecovery speed, stream recovery distribution, and stream load balancing.Methods for failure recovery range from “hot-standby” (where a stream isplaying to RAM but not being transmitted to the network interface) wheretwice the bandwidth is consumed so that recovery from a fault is asquick as possible to “frozen-standby” where speed is sacrificed overongoing bandwidth consumption. The goal is to choose the “coldest”standby model that sufficiently addresses a streaming client recoveryspeed requirement as it will greatly reduce the resource requirement.

Recovery distribution deals with which server is responsible formaintaining the recovery data and instantiating the recovery process.Two methods are usable for resiliency enforcement—resiliency partner(one to one—or assigning responsibility to one other machine) andresiliency group (one to many—or distributing responsibility acrossseveral other machines). Although there is little difference in resourceconsumption, a resiliency group offers slightly quicker recovery (sinceseveral machines are recovering from the loss of both active and“standby” streams in parallel) as well as some level of recovery shouldmore than one failure be experienced (since all recovery data is not inone machine).

Finally, load balancing deals with distributing active streams across astream group in a fashion that attempts to balance overall system loadwith cache effectiveness. The load balancer must not only deal withwhere to host a new stream, but it must also deal with changes, bothplanned and unplanned, in system resources. In a “hot-standby” model a“standby” stream consumes just as many resources as an active stream andtherefore must be actively load balanced from the beginning. As therecovery speed is pushed more toward the “frozen-standby” model, therequirement to actively load balance “standby” streams diminishes.

In an implementation of stream resiliency, the system offers multiplestream resiliency options which allow the system administrator tobalance recovery speed and hardware consumption.

8. Load Balancing and Stream Handoff in Trick Mode

In addition to the core load balancing and stream handoff algorithms forthe stream itself, additional mechanisms are defined for the trick modes(e.g., rewind, fast rewind, pause, fast forward) associated with thestreams. When a trick mode is invoked, the dynamic load balancing andresiliency mechanisms of this invention will be applied to the trickmode stream. When trick mode is not invoked, the load balancing andresiliency mechanisms will not be applied to the trick mode stream. As aresult of this enhancement, cache efficiency is significantly enhancedbecause trick mode streams are pulled into a fewer number of streamservers, resulting in higher reuse rates.

9. Stream Handoff Based on Configuration Change

In addition to the stream handoff algorithms being invoked based uponfailure of a member of a stream or vault group, they can also be invokedbased upon a configuration change. In one case the configuration changeoccurs because of an explicit operator action. In another case theconfiguration change occurs based upon device failure. In either case,the same algorithms are used to handoff streams in a fault tolerantfashion.

10. Proxy Content Vault Handle Management

In addition the maintenance of separate internal and external handlesfor a logical stream group, a vault server can also implement the samemechanism to hide the internal architecture from external entities, andto enable transparent fault tolerance. A handle that is exposed to abackoffice client of the vault server is remapped to an internal handleif failover occurs and the content is moved to a different vault. Inaddition the mechanism of using heartbeats for keepalive detection, andelection of masters within a vault group, is also implementable within avault group.

F. Server Resiliency Protocol

1. Server Resiliency Group Composition

The purpose of a server resiliency group is to preserve state of groupmembers so that if one member fails another member can takeresponsibility of redirecting the load from the failed member.

A server resiliency group is composed of two or more servers which haveentities of the same type. That is, a stream server will be in aresiliency group with stream servers. A server with both a stream serverand vault server functionality can be members of two resiliency groups.

2. Server Resiliency Protocol

According to the server resiliency protocol, the payload of packets isencoded in self describing units. Each unit contains a length, anoperation code and operation data. A packet contains at least one unitand units are not split across packet boundaries. When a packet fills,it is sent. Otherwise it is timed to make sure it waits no longer than aset period (e.g., 1-5 ms locally—a metro level timer may be longer).This timer influences; 1) the delay in determining that a server hasfailed, and 2) the recovery time. Units are added on an event basis.

3. Operations

Operations may be one of three types:

-   -   Request (.req)—These operations request response information.    -   Response (.rsp)—These operations respond to a previous request.    -   Announcement (.ann)—These operations announce state information.

4. Header

The protocol header contains 1) a Sequence Number, 64 bit unsigned andassumed to never wrap; and 2) Time—64 bit unsigned—assumed to neverwrap—in 100 microsecond ticks—estimate of transmit time. This estimatemay need to account for queued data on adapter unless there is very lowtransmit jitter. This time is used by recipients (remember this ismulticast) to develop a model of transmitter jitter. The jitter model isused to determine timers for detecting server failure and packet loss.

Operation Name: Enter.ann

Function: Announce a new member of the resiliency group is entering (thesource MAC)

Parameters: None

Action: Upon receipt any previous state about the source MAC isdiscarded (e.g. sequence number) and the source is added to the locallist of group members.

Operation Name: Rename.ann

Function: Announce a new member of the resiliency group is entering (thesource MAC)

Parameters: Member—MAC address previously used by group member

Action: Upon receipt any previous state about the source MAC replacesMember in the local list of members. All additional state information isleft unchanged. When a port fails this is used to switch to another portand obviously would be the first unit in the first packet sent after thechange.

Operation Name: Leave.ann

Function: Announce a member of the resiliency group is leaving (thesource MAC)

Parameters: None

Action: Upon receipt any previous state about the source MAC isdiscarded (e.g. sequence number) and the source is deleted from thelocal list of group members.

Operation Name: Retransmit.ann

Function: Request the retransmission of apparently dropped packet(s)

Parameters: Member—MAC address of member

-   -   Dropped—16 bit unsigned—Count of sequence numbers to retransmit    -   Sequence—Array of 64 bit unsigned—Sequence numbers to retransmit        Action: If Member is a local MAC address then the request        packet(s) should be retransmitted.

Operation Name: Receive.ann

Function: Announce the reception of all packets up to the specifiedsequence number

Parameters: Member—MAC address—MAC address of member

-   -   Sequence—64 bit unsigned—Sequence number of last packet received        in order        Action: If Member is a local MAC address then the all packets up        to Sequence are acknowledged by the source MAC address. Once        acknowledged by all group members the transmit buffers may be        reused.

Operation Name: State.ann

Function: Announce state from the member

Parameters: Entity ID—16 bit unsigned—ID for source entity

-   -   Type ID—16 bit unsigned—ID of state type for entity    -   Length—16 bit unsigned—Length in bytes of state    -   State Data—byte array with length specified by Length—the state        data        Action: Upon reception the state resiliency handler for the        entity referenced by the entity ID is call with the Type ID,        Length and State Data. The semantics of the type is left up to        the entity. The State Data is opaque except to the state        resiliency handler.

G. Streaming Resiliency

1. Stream State Resiliency

In an embodiment, stream state is preserved across stream server fault.This is done through two logging mechanisms; 1) logging to resiliencygroup, and 2) logging to disk. The state logged is the same regardlessof whether it is logged to a resiliency group or to disk. A streamerwill log updates to its state to both locations. The other members ofthe resiliency group maintain a “shadow” server state which can be usedwhen a stream server failure is detected to handoff the impacted streamsto other stream servers. Potentially the other resiliency group memberscan also log this “shadow” state to provide another level of resiliency.

2. Logging to Disk

In one class of solutions a fairly full function database can be used toupdate and log stream server state. In another preferred class ofsolutions a lower level of functionality can be used to achieve higherperformance. Some nodes have a relatively modest number of streams perdevice measured in thousands with a definite maximum. The most commonstream state changes (rewind, fast forward, pause) are non-criticalindividual transactions and have a built-in retry from the human holdingthe IR remote.

In an embodiment, no database is used and instead a flat file is createdwhich is simply a copy of the memory image. When the full state iswritten out to disk, it is merely copied in memory and then written witha lock being held on the actual data structures only during thememory-to-memory copy.

The model is that the streams persist forever. They are updated, markedactive, valid, etc. and those transactions are logged. The most commontransactions should be very short log entries (for example, 32 bytes orless) so that the total amount of disk I/O is relatively low. A shorttimer is used to force out the log on a periodic basis, e.g. every 100ms. The full state is written opportunistically but should be written atleast every minute or so to reduce log file size.

Every entry in the log contains an integrity vector (for example, use a64 bit XOR of the entry) and a guaranteed monotonic transaction ID(pretend the log went on “forever” and use the 64 bit integer byteoffset of the log entry in the “virtual log file” as the transactionID—this makes the transaction ID dependent on the sequence of logentries over all time and creates an easy unique validation independentof the integrity vector). Each stream entry in the full state write todisk contains the Transaction ID of the last logged update and anintegrity vector for the stream entry including the Transaction ID.

The file level serialization is that a log file cannot be deleted untilthe full stream state has been successfully written out incorporates allof changes from the log file.

In order for the full stream state to be disk controller resilient thestate file and log files are written in redundant locations to survive acontroller failure.

To recover the stream state, one would read the latest state file andverify the integrity vector of each entry. Then one would read the logfiles and apply the changes if and only the Transaction ID of the logentry was greater than the Transaction ID in the data structure. If avalidation check fails the other copy of the file is used. With everyentry having multiple ways to validate the robustness of this recoverycan be quite great.

Any needed indices would then be built (e.g. just a hash or two) and thesystem is ready to be activated. A possible optimization is to store thestream entries as a local cache to reduce remote lookup of informationwhen starting another copy of the same content (this creates the needfor a stream state “valid” while the stream is not active but beingretained as cached state).

3. LSCP Connection State Resiliency

It is possible if the STB code is robust enough that LSCP state could bepreserved across stream server faults very easily, however, this wouldbecome a requirement for every client on every STB. In an embodiment,the stream server responds to ARP's for the stream control IP addresswith either a multicast previously registered via GMRP with the localswitch or a unicast which been configured to a VLAN containing allmembers of a stream group but has not been resolved to a port. Thisapproach assumes all stream group members are reachable via layer 2addressing.

Once done, this will imply all stream servers will see all controltraffic.

At this point a new layer is added between the MAC and the TCPstack—which is called NAT (Video Network Address Translation). The VNAThas the classical NAT functions of address translation and sequencenumber translation, etc. It does not have to do some of the more painfulNAT functions such as finding and translating embedded addressed.However, it does have multiple responsibilities beyond a regular NAT. Itmust be able to: Decide whether inbound are actually destined for thisstreamer or not:

1. Spoof TCP connection setup and tear down

2. Maintain distributed state for retransmissions across a streamerfailure

3. Handle retransmissions if needed

In an embodiment, state information includes state information relatedto a particular stream or streams. For example, state informationrelated to a stream may include a stream descriptor, stream identifier,a time offset (for example a time offset from real-time relative toother streams carrying the same digital video content and or a timeoffset relative to the ingest time of the digital video content), and adestination IP and or MAC address for the stream. A time offset for astream can occur as the result of initiating a trick mode (e.g., rewind,fast rewind, pause, fast forward).

Although specific embodiments of the invention have been described andillustrated, the invention is not to be limited to the specific forms orarrangements of parts as described and illustrated herein. The inventionis limited only by the claims.

1. A method for managing the streaming of digital video content tomultiple clients, the method comprising: identifying an attribute of acontent element that is streamed to a client from a stream server; andselecting a protection mechanism for the content element as a functionof the attribute, wherein the protection mechanism enables streaming ofthe content element to the client from another stream server in theevent of a resource failure; wherein selecting a protection mechanismfor the content element comprises: mapping attributes of contentelements to a set to categories such that an attribute falls within acategory of the set of categories; and mapping each category of the setof categories to a protection mechanism; wherein the combination of thetwo mappings creates an indirect mapping between attributes of contentelements and the protection mechanisms.
 2. The method of claim 1 whereinthe attribute of the content element is an indication of the type of thecontent element.
 3. The method of claim 1 wherein the attribute of thecontent element is an indication of the importance of the contentelement.
 4. The method of claim 3 wherein the selection of a protectionmechanism further comprises selecting a protection mechanism thatprovides a higher level of protection for a more important contentelement and a lower level of protection for a less important contentelement.
 5. The method of claim 1 wherein a selected protectionmechanism comprises: providing the same digital video content to firstand second stream servers in a logical group of stream servers; storingthe digital video content into random access memory (RAM) at both thefirst and second stream servers; streaming the digital video content toa client from the first stream server but not the second stream server;and switching to streaming the digital video content to the client fromthe second stream sever instead of the first stream server after afailure is detected in the first stream server, the steaming includingactivating an interface of the second stream sever that was notactivated until after the failure in the first stream server wasdetected.
 6. The method of claim 1 further comprising prioritizingmultiple attributes and associating each of the prioritized attributeswith a protection mechanism that represents a different level ofprotection.
 7. The method of claim 6 further comprising associatinghigher priorities with protection mechanisms that provide higher levelsof protection and associating lower priorities with protectionmechanisms that provide lower levels of protection.
 8. The method ofclaim 1 wherein an attribute of a content element is an indication ofthe popularity of the content element and wherein the popularity of acontent element is determined based on real time popularity information.9. The method of claim 1 wherein identifying an attribute comprisesgenerating popularity information related to the content element. 10.The method of claim 1 wherein selecting a protection mechanismcomprises: identifying the resource requirements of multiple protectionmechanisms; establishing a fault recovery requirement for a contentelement; and selecting, for the content element, the protectionmechanism with the lowest resource requirement that meets the faultrecovery requirement for the content element.
 11. The method of claim 1further comprising establishing a logical group of stream servers anddistributing streams associated with the same content element amongstthe stream servers in the logical group as a function of the popularityof the content element.
 12. The method of claim 1 further comprising:establishing a logical group of IP-connected stream servers; generatingstate information at multiple stream servers within the logical group;sharing state information amongst stream servers in the logical group;and using the shared state information to respond to resource failureswithin the logical group.
 13. A system for managing the streaming ofdigital video content to multiple clients, the system comprising: aprocessor; and memory for storing processor executable instructions,which when executed by the processor perform the functions of: anattribute engine configured to identify an attribute of a contentelement that is streamed to a client; and a protection mechanismselector configured to select a protection mechanism for the contentelement as a function of the attribute, wherein the protection mechanismenables streaming of the content element to the client in the event of aresource failure; wherein selecting a protection mechanism for thecontent element comprises: mapping attributes of content elements to aset to categories such that an attribute falls within a category of theset of categories; and mapping each category of the set of categories toa protection mechanism; wherein the combination of the two mappingscreates an indirect mapping between attributes of content elements andthe protection mechanisms.
 14. The system of claim 13 wherein theattributes are related to an indication of the popularity of the contentelements and the set of categories is related to priority such that acontent element of a given popularity is mapped to a specific priorityand wherein each priority is mapped to a specific protection mechanism.15. The system of claim 14 wherein the selection of a protectionmechanism further comprises selecting a protection mechanism thatprovides a higher level of protection for a more popular content elementand a lower level of protection for a less popular content element. 16.A system for managing the streaming of digital video content to multipleclients, the system comprising: multiple stream servers that arelogically connected into a logical group, wherein at least one of thestream servers in the logical group comprises: an attribute engineconfigured to identify an attribute of a content element that isstreamed from one of the stream servers in the logical group to aclient; and a protection mechanism selector configured to select aprotection mechanism for the content element as a function of theattribute, wherein the protection mechanism enables streaming of thecontent element from one of the stream servers in the logical group tothe client in the event of a resource failure; wherein selecting aprotection mechanism for the content element comprises: mappingattributes of content elements to a set to categories such that anattribute falls within a category of the set of categories; and mappingeach category of the set of categories to a protection mechanism;wherein the combination of the two mappings creates an indirect mappingbetween attributes of content elements and the protection mechanisms.17. The system of claim 16 wherein the attributes are related to anindication of the popularity of the content elements and the set ofcategories is related to priority such that a content element of a givenpopularity is mapped to a specific priority and wherein each priority ismapped to a specific protection mechanism.
 18. The system of claim 17wherein the selection of a protection mechanism further comprisesselecting a protection mechanism that provides a higher level ofprotection for a more popular content element and a lower level ofprotection for a less popular content element.
 19. The method of claim 1wherein the attributes are related to an indication of the popularity ofthe content elements and the set of categories is related to prioritysuch that a content element of a given popularity is mapped to aspecific priority and wherein each priority is mapped to a specificprotection mechanism.
 20. The method of claim 1 wherein the attributesare related to an indication of the type of the content elements and theset of categories is related to priority such that a content element ofa given type is mapped to a specific priority and wherein each priorityis mapped to a specific protection mechanism.
 21. The method of claim 19wherein protection mechanisms that provide higher levels of protectionare associated with categories of more popular content elements andwherein protection mechanisms that provide lower levels of protectionare associated with categories of less popular content elements.